Robots.txt Generation
The robots.txt file tells search engine crawlers which pages they can access. When enabled, Reverb 2.0 copies a template robots.txt to your output directory and completes its Sitemap: line with the Sitemap Base URL, because crawlers require a full URL there.
Default robots.txt Content
The default template allows all crawlers to access all content and points them to the sitemap. With the Sitemap Base URL set to
https://docs.example.com/help/, the generated file reads:User-agent: * Allow: / Sitemap: https://docs.example.com/help/sitemap.xmlWhen Sitemap Base URL is empty, or Generate Sitemap is disabled, the
Sitemap:line is left out.Customizing robots.txt
To customize the robots.txt file, create a Format Override:
In ePublisher Designer, expand your Reverb 2.0 target
Navigate to Format > Pages > robots.txt
Right-click and select Create Override
Edit the file with your custom rules
ePublisher copies your rules as written. A relative
Sitemap:line, such asSitemap: sitemap.xml, is completed with the Sitemap Base URL the same way as in the default template. An absolute or protocol-relativeSitemap:line is kept as is.